Discriminative vs Informative Learning

The goal of pattern classification can be approached from two points of view: informative where the classifier learns the class densities, or discriminative where the focus is on learning the class boundaries without regard to the underlying class densities. We review and synthesize the tradeoffs between these two approaches for simple classifiers, and extend the results to modern techniques such as Naive Bayes and Generalized Additive Models. Data mining applications ofI^_ -----A.-:LL2--.-s.. ,cL:,L ,l:m,no:,nn, &..,+..ron IJeLl uyaraw 1u IJUI: U”Lllcz,U “I Lup uLLlL~~DI”II(III IGi(lr”UIG.3 where the tradeoffs between informative and discriminative classifiers are especially relevant. Experimental results are provided for simulated and real data.’ KDD and Classification Automatic classification is among the main goals of data mining systems (Fayyad, Piatetsky-Shapiro, & Smyth 1996). Given a database of observations consisting of input (predictor) and output (response, i.e. class label) variables, a classifier seeks to learn relationships between the predictors and response that allow it to assign a new observation, whose response is unknown, into one of the K predetermined classes. The goal of good classification is to minimize misclassifications or the expected cost of misclassifications if some types of mistakes are more costly than others. Classifiers can be segmented into two groups: 1. Informative: These are classifiers that model the ~lslaa rlancit;cl~ f’laraifirn.t.inn in done hv nun.mininp “A~“” UUAIYAYA”“. d,,,,...,,.,,.-IJ ---I---------o the likelihood of each class producing the features and assigning to the most likely class. Examples include Fisher Discriminant Analysis, Hidden Markov Models, and Naive Bayes. Because each class density is considered separately from the others, these models are relatively easy to train. ‘Copyright 01997, American Association for Artificial Intelligence (www,aaai.org). All rights reserved. 2. Discriminative: Here, no attempt is made to model the underlying class feature densities. The focus is on modeling the class boundaries or the class membership probabilities directly. Examples include Logistic Regression, Neural Networks, and Generalized Additive Models. Because this requires simultaneous consideration of all other classes, these models are harder to train, often involve iterative algorithms, and do not scale well. lated via Bayes rule, but often lead to different decision rules, especially when the class density model is incorrect or there are few training observations relative to the number of parameters in the model. There are tradeoffs between the two approaches in terms of ease of training and classification performance. Precise statements can only be made for very simple classifiers, but the lessons can be applied to more sophisticated techniques. In this paper we review the known statistical results that apply to simple non-discriminative classifiers and we demonstrate how modern techniques can be categorized as being discriminative or not. Using Naive Bayes and GAM applied both to simulation and real data, we exemplify that, counter-intuitively, discriminative training may not always lead to the best classifiers. We also propose methods of combining the two approaches. We focus on parametric techniques although similar results obtain in the non-parametric case. With the advent of increasingly sophisticated classification techniques, :c :n :-s.r\rtnn+ +n znnl:n,,‘*z.,ha+ nqtomrr,.., th,, ~1nnn;fic.r Ib 13 ML~“llJallb IJ” lGall*G, NyIIcb” bauzEj”LJ “A&G CIIclJUUIII~L falls in, because the assumptions, problems and fixes for each type are different. Overview of Bayesian Classification Theory Formally, the classification problem consists of assigning a vector observation 2 E 7V into one of K classes. The true class is denoted by y E { 1, . . . , K}. The clas-